Import necessary packages¶
In [502]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import os
import pandas_profiling
pd.set_option('display.max_columns',None)
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from scipy.stats import zscore
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn import metrics
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
Listing and evaluating each file¶
In [503]:
for f in os.listdir():
print(f.ljust(30) +"--" + str(round(os.path.getsize(f) / 1000000, 2)) + 'MB')
In [504]:
#No Additional files present apart from data set
Importing csv to data frame¶
In [505]:
df=pd.read_csv('vehicle.csv')
df.shape
Out[505]:
In [506]:
#contains 864 rows and 19 columns
df.head()
Out[506]:
In [507]:
df.tail()
Out[507]:
- Values looks distributed at first glance
- 18 independent variables (compactness, circularity, distance_circularity, radius_ratio, pr.axis_aspect_ratio, max.length_aspect_ratio, scatter_ratio, elongatedness, pr.axis_rectangularity, max.length_rectangularity, scaled_variance, scaled_variance.1, scaled_radius_of_gyration, scaled_radius_of_gyration.1, skewness_about, skewness_about.1, skewness_about.2, hollows_ratio) and one dependent variable (class).
- All columns are numeric except dependent column 'class'
Exploring Data¶
In [508]:
#Intial look on table columns and types and non null counts
df.info()
- Total Columns: 19
- Target column: class (object Dtype) - contains details on cars/vans/bus
- 14- float field and 4 int fields
- total 846 rows
- few null entries on multiple fields - need to identify the pattern and fix it
- No special charcters in numeric fields else we migh have noticed other dtypes.
In [509]:
#understading the data
df.describe(include='all').T
Out[509]:
- Class has 3 categories and max freq is 429
- numeric fields have huge variation from field to field, need to standardize the data for better outcomes
- mean and median are close by for majority of the fields hints the fields are close to normal distributions
In [ ]:
Null Value Analysis & Treatment¶
In [510]:
#identifying null values
df.isnull().sum()
Out[510]:
- Multiple Null values found on the data set
- We didnt find any column to be any other format apart from Int or float, which suggests there are possibly no un related symbols or texts in the data set
In [511]:
#replace null values as nan
df = df.replace(' ', np.nan)
In [512]:
#Replacing the missing values by median
# median are more robust towards the data distribution compared to mean hence selected median for replacement
for i in df.columns[:17]:
median_value = df[i].median()
df[i] = df[i].fillna(median_value)
In [513]:
df.isnull().sum()
Out[513]:
In [514]:
#revalidating info - for post changes details
df.info()
- Rechecking info, all columns has same number of non-null count
In [ ]:
In [ ]:
Outlier Analysis & Treatment¶
In [515]:
#understanding outliers
df.boxplot(figsize=(35,20))
Out[515]:
- Multiple outliers were present in the cols such as - radius_ratio,pr_axis_aspect_ratio etc
- Will treat the outliers with replacing it with medians/max boundry value ( IQR * 1.5), we might miss-represent few data points for outliers but our main goal is to generalize the model for better overall prediction.
- Clustering techniques are hihgly volatile towards outliers and need treatment.
- We might observe new outliers once we treat the data as values will be updated and new IQR will be formed, but we will treat the data set only once with it.
In [516]:
#understanding distribution before the outlier treatment
df.hist(figsize=(35,20));
- as observed in box plot, we can clearly see many columns has long tails in histogram as well
- Majority of the columns data are distributed but having multiple peaks
Outlier Treatment¶
- we will be treating the columns with same logic used in box plot using IQR
- We are replacing the Outliers with boundry values to keep the higer values colse to higher and lower boundries.
- Goal is to keep the value inside acceptable ranges without loosing its main charatcteristic of being outlier or being a higher/lower number. Using this approach we will slightly increase the frequency on boundry values but they will be inside range.
- Replacing with Mean or median might miss represent the main characteristics of the data set which is pushing it to be be outlier.
In [517]:
#- selecting all but leaving out the last columns whihc is our target
for cols in df.columns[:-1]:
q1=df[cols].quantile(0.25)
q3=df[cols].quantile(0.75)
iqr=q3-q1
low=q1-1.5*iqr
high=q3+1.5*iqr
df.loc[(df[cols]<low),cols]=low
df.loc[(df[cols]>high),cols]=high
In [518]:
#reevaluate the box plot for outliers
df.boxplot(figsize=(35,20))
Out[518]:
- we can notice exisitng outliers were gone but fortunately no new new outliers were imerged in any columns
Understanding the Attributes¶
In [519]:
#understanding Dependent column
sns.countplot(df['class']);
plt.show()
print(df['class'].value_counts())
- Majority of the data set is having cars (429) followed by bus (218) and van (199)
In [520]:
#have to use label encoding for class to get a view on correlation
#using labelencoder and not one hot encoding for not increasing classes at this point
from sklearn.preprocessing import LabelEncoder
lblEncoder=LabelEncoder()
df['class']=lblEncoder.fit_transform(df['class'])
In [521]:
cor_df=df.corr()
In [522]:
fig,ax=plt.subplots(figsize=(15,15));
sns.heatmap(cor_df,annot=True);
- Class columns doesnt have very high correlation with any of the columns
- On the other hand elongatednes has very high correlation with multiple columns, we can plan for it to be removed to reduce multicoliniarity.
- There are multiple columns which have high correlation with other columns like circularity with max.length_rectangularity & scaled_radius of gyration, scatter_ratio with scaled_variance.1
- We can identify columns which has less correlation with class col and plan it for removal down the line.
In [523]:
#Viewing all details
sns.pairplot(df)
Out[523]:
In [524]:
#viewing with class distribution on each graph
sns.pairplot(df, hue='class')
Out[524]:
